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COMPUTER BASED SYSTEM FOR MANXPUXATING DIGITAL MEDIA 
Technical Field 

This invention rdates to a computet software system for manipulating digital media. 
^Background Axt 

5 Application software for editing digital video is an exttetocly sophisticated and powerful tool 
because it is primarily designed for, and sold to, the video professional. Such an, individual 
requires access to many complex functions and Is prepared to invest rime and effort in 
learning to become skilled in their use. Historically, the terminology and conventions of 
Digital Editing have evolved from a traditional 61m editing environment where rushes arc cut 
10 and spliced together to tell a story ot follow a script* As digital mixer technology advanced 
new techniques were combined with these conventional methods to form the early 
pioneering software based digital editors. 

To the video or film professional editing is second nature and the complexities of a time- 
based media go unnoticed since, having already grasped concepts and learned processes, they 
15 are able to concentrate on the nuances of different editing packages, of which there are 
many. 

Conventionally these packages, through the use of a Graphical User Interface (GUI), Attempt 
to provide on abstraction of the media m terms of many separate trades of video and audio. 
These arc represented on the output device in symbolic fashion and provision is made for 

20 interacting with these representations using an input device such as a mouse. Typically the » 
purpose is to create a new piece of media as an output file, composed by assembling dips or 
segments of video and audio along a timeline that represents the temporal ordering of 
frames. Spcdal effects such as wipes and fades can be Incorporated, transparent overlays can 
be add e d, colour and contrast can be adjusted The list of manipulations made possible by 

25 such tools is very long indeed. A typical system is described in., for example, Foreman; Kevin 
J,, et, sJ, "Graphical user interface for a video editing system", U.S. Patent. 6,469,711. 

It is possible, however, that an individual who .Is a consumer of media, rather than a 
producer, may need to perform a simple editing operation, on A. media file in order to 
accomplish their primary task; for example to give a multi-media presentation. In this cree 
30 such tools have thdr drawbacks. They may be too expensive to justify individually, or to 
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have enough of in order to be available when or ^h.exe needed. The limited amount of use 
and fee small ftaction of the capabilities used in such situations may make ihem uneconomic. 
The steep kaxnjng curve associated with such tools may mean that an inappropriate amount 
of effort is expended on something that Is not the pdmary occupation or concern of the tool 
5 user. For occasional or infrequent use there wll be reluctance on the pa*t of any user 
repeatedly to swtch environments or learn and releatri new tools to perform simple last 
minute tasks. 

This situation parallels previous well-known situations where Improvements in the 
availability, usability and price/perfbtmance ratio of consumer IT equipment, has caused a 

10 significant reappraisal of what is possible and ft change in behaviour to exploit new 
possibilities. For example, the production of high-quality printed documents was once the 
province of highly skilled people using expensive and specialised equipment, Now anybody 
vnth a need to produce such a document, who has access to a computer and a word- 
processing program, can do so. A similar shift in paradigm is happening In Digital Video 

15 Editing, vrfrere there is a need for highly accessible and usable tooJ.8 that focus on the needs 
of a new generation of user, and not necessarily try to recreate the feel of a traditional, video 
editing envicoofieent, 

Such took must exhibit an intuitive, predictable and consistent behaviour as understood by a 

new generation of digital media professionals who may \pu11 bo extremely ft mi l iar with the 
20 manipulation of documents of various kinds through a computer's GUI, but be completely 

unfamiliar xnth the characteristics of time-based media, The tools need not supercede long 
established and specialised tools used by trained professional but, rather, provide a bridge in 
order that M users may be as comfortable woddng 'with time based media as they are 

-worldng with documents. 

25 Conventionally, video editors are structured as specialised 'monolithic' applications. Current 
software technology, however, is well capable of adding sophisticated editing functions to 
unrelated applications through the use of software •plug-ins'- The Microsoft® DircctShow® 
Editing Services is an application programming interface (API) that is built on top of 
Microsoft® DirectShow® that allows video editing capabilities to be added to applications. 

30 In this example filters, implemented as Common Object Module components, are created 
and Interconnected to form filter graphs* As another example the QuickTime track based 
architecture i* the foundation of many modem day editors such as Adobe Premiere. It offers 
embedded API based access, resident below the application layer that provides for simple 
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track noanipularion. There is no reason why such plng-ins may not be deployed in 
applications that are not, primarily, designed for video editing. 

Conventionally* video editors axe application programs that run on high-end PCs and 
5 workstations, under desktop-oriented operating systems such as Microsoft Window or 
Apple's Mac OSX, often with high-resolution screens and Mgh-bandwidth network 
connectivity. The viewing of media flies, however, can take place on an ever-expanding list 
of devices with many different capabilities, such as laptops, mobile PDAs with wireless 
OTflU&eaSvity, mobile phones, set-top boxes and hard-disc based personal video recorders 
10 (PVRs). The concept of a simple media manipulation tool integrated into the media playet 
component is as relevant in these cases as it is in that of the standard PC, possibly tnotc so 
since, for example, a PVR may not have a run-time environment capable of running external 
applications such as video editors. 

15 Accordingly, these arc the attributes of a tool that is more appropriate to the. needs of such a 
consumer. 

1. Sim p le and intuitive to use; in particular, little time and effort is rexjuired to learn 
enough to accomplish the task in hand, 

2- Terminology and workflow consistent with a shift in convention towards action led 
20 digital media editing; e.g. ^og* to select as with modern VCR's and a simple crop 

ability to trim the running length of a piece of media. 

3, Available whenever and wherever needed, even If the user did not foresee the need 
fot such a tool until that need cropped up, i.e., the edit capability is provided as an 
intrinsic part of the environment by way of any player of the media, 

25 4. Provides a consistent interface to the user irrespective of the type of 'container 1 

application it is associated with. It would look exactly the same whether incorporated 
Into an electronic text document, t? spreadsheet; or a slide presentation. 

5. Persistence of modifications; e.g. the user opens media object, ox a document 
containing an embedded media object and expects any changes they make to persist 
30 " between sessions. 
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Dia closure of Invention 

The invention relates to a method for adding the capability foe media-manipulation to 
softwxe media players, such that this capability is intrinsic to the media player, and which 
comprises a set of tools that ensure that consistent behavioural, visual and functional aspects 
5 axe maintained between media player applications* 

Briefly;, fee invention -works as follows. 

According to one aspect of the invention a Graphical User Interface (GUI) for editing is 
implemented. 

In the preferred embodiment of this aspect of the in^endon a plug-in module is loaded into 
10 the computer's memory to provide the specific functionality required, This software module 
has interfaces to a media delivery subsystem, such as the Microsoft® DircctSho-w® 
architecture for the Microsoft® Windows® platform that provides services for streaming, 
buffering, synchronisation, decoding and rendering of video and. audio. Media is streamed 
into a local c*che -that provides for fine-grain scrubbing 'jog' and looping 1 of short sections 
15 around in and out points. A set of instructions is devised for each piece of media and. its 
interaction -with a timeline. Specific dements are constructed in memory to process these 
instructions and subsequently handle the media to a suitable form, as compatible with die 
media play architecture in operation. New and modified dements maybe constructed and 
reconstructed as required, each clement may process but is not limited to a single set of 
20 instructions or piece of media. 

The Biocdonality provided by this software module consists o£- 

• Graphics rendering to allow combination and/or overlay of graphical data for die GUI 
with pixels that ate decoded fiom the video part of the media file and rendered Into the 
video wea on the screen. 

25 

■ A cache foe portions of the media file in the memory of the client machine. 

■ A state machine, whose transitions guide ft user through, a sequence of interactions with a 
graphical user interface (GUI), 

30 

• Graphics that implements visual feedback of the current state to the user, 
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• Graphics that implements a visual metaphor that provides the user with an intuitive 
understanding of the operation of the interface. 

■ An exports* fgr the persistence of the chosen manipulations, for example; to "Save" the 
5 processed media to memory creating a new media object or create a new set of 

instructiona that describes the precise operation required to effect the manipuktions for 
playback. Including but not limited to, such instruction as references to sections within a 
remote piece of media. 

10 • Graphics that allows labels of various types to be added to significant parts of the fftedia 
file in order to identify them as such, 

• Graphics that allows the definition of actions to be taken when significant parts of the 
media file are encountered in notmal playback. 

15 

In this embodiment of this aspect of the invention the GUI is provided by modules within 
the software ftamework chat implements title media player, by the addition of visible user 
interface components (buttons, text boxes, etc.) either overlaid or actually burnt into the 
tendered video window (Lc* the pixels written to the framestore by the video renderer are 
20 overwritten). In the Windows Media® architecture, where software filter gtaph components 

are linked together to implement a media player, this functionality may be added Into a video 

renderer filter or an overlay filter. 

In another embodiment of this aspect of the Invention the GUI is provided by software 
modules, other than those embedded within the media player fisuneworlt, such as ActiveX 
25 controls. 

According co another aspect of the invention elements are exchanged between instances of a 
media player. 

In the preferred embodiment of this aspect of the invention the VPIndows© Media 
environment is employed such that oae instance of the player may be used to manage the 
30 "master" timeline, while another allows clips to be trimmed to the desired length and, then 
dragged and dropped into the "master" player instance. At this rime the recipient instance 
may chose to combine die filter graph for die new piece of media wfth. those jjready in 
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existence, Ot it m*y chose to reconstruct a new filter graph based on the complexity and 
requited interaction of the current timeline objects, 

According to another aspect of the invention a process flow is provided that provides for 
untrained uscars to achieve their goal with minimum effort, and distraction from theit primary 
5 task. 

In the prefaced embodiment of this aspect of dlO invention state machines help walk the 
users through operations to avoid mistakes and distil the complexity of editing into bounded 
and easy to understand pa?ocesses. Visual and tactile feedback vritl provide rapid confidence 
In the task ond aid progress; eg. to slim down a media object the user will select a "Start 
10 Here" in point and guided towards a "Stop Here" out point 

Effective confirmation methods are employed to inform and protect the actions of the user a 
visual metaphors will be provided from the embedded editor level to identify nodes of the 
current state machine. For example, the video window may show a fflmstap with the current 
frame highlighted subsequent frames notrnal and previous frames, Le* those cropped, with, ft 
15 strike out market. 

According « another aspect of the invention meta-data in the media, file h recognised by the 
system and used as a sttcom of control information chat is used to assist editing operations. 

In one embodiment of this aspect of the invention the meta-data may include but is not 
limited to: 

20 * Tlmecodc, 

• Closed caption 

• Edit points used during the creation of the media, 

• Fotmat-depepdent properties such as GOP boundaries in MPEG, 

• Data generated as a result of post-processing such as shot change information. 

25 

The control information identifies sigiificftnt points jn the media and triggers events chat 
cause instructional ot informative information to be displayed* For example dialogue boxes 
may pop up during playback with labels such as "Start Here" (EN) or "Stop He*e" (OUT) . 

30 According to another aspect of the invention the media player with intrio.eic media* 
manipulation capability runs on a number of platforms of different types, configurations and 
capabilities. 
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111 one embodiment of thte aspect of die invention the media player runs on PCs and 
workstations, 

S In another embodiment of this aspect of lie invention the media player runs on set-top 
boxes. 

In another embodiment of this aspect of the invention the media player runs on Personal 
Video Recorder (PVfts). 

10 

In another embodiment of this aspect of the invention the media player runs on mobile 
devices including Personal Digital Assistants PDAs. 

In another embodiment of this aspect of the invention the media player runs on mobile 
15 phoned. 

According to another aspect of the invention the visual appearance of the GUI is sensitive to 
the context in which the user of the system is working in order that the tools may be non- 
intrusive when not needed, but available when called for. 

In the preferred embodiment of this aspect of the invention the visibility of tie GUI is 
20 dependent on whether ot not the -cursor, falls outside or inside the video window, If the 
cursot is outside tfce window the controls ace invisible and disabled; if the cursor is inside the 
window the controls are visible and enabled 

Brief Description Of Drawings 

Figure 1 shows the Graphical User Interface (GUI) of a standard media player Th& usual 
25 selection of menu bats, buttons and sliders can be seen to surround the rectangular video 
•window 10. A bumt-to tirrjecode 11 and logo 1Z arc visible at the top of the video window 
10. 

Figure 2 shows the same player. z$ shown in Figure 1 but with the intrinsic editing GUI 
activated and the controls visible within the video window. In this particular instance the 
30 GUI is activated by moving the mouse pointer from the outside of the video window to an 
appropriate region inside. 
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At the bottom an edit bat 2l is visible that represents the timeline of the loaded media file 
togot&er vdth a pointer 22 that indicates the current position Nvitbia the file. Above and to 
die left of tie edit bar are left and sight "brace" symbols 23 that identify buttons for 
specifying "in" and "out" points, Respectively. To the right of these there is a button 24 for 
5 performing die "take clip" operation, the symbol for which has a bar through it^ meaning 
that the button is not active because no ,l in" and "out" points have as yet been set. To die 
rigfrt of due video window is a text box 25 with SMPTE timecode visible, and a "seek" 
button 26. Typing a fimecode into this box and pressing "seek" causes a peek to the 
appropriate frame with this timecode labdL 

10 Figure 3 shows the situation when the °in point" button 31 has been pressed* The button 
has changed colour to indicate the change of state and a green handle 32 has appeared inside 
the edit bar to indicate the presence and position of title in-pofot. 

Figures 4 and S show the next step in this process; in Figure 4 the video has been moved 
on to locate the out-point and in Flgute 5 the out-point button SI has been pressed, so 
15 causing die "tafoc dip" button to become active find a red out-point handle 53 to appear in 
the edit box,. Both die in and out-point handles may be dragged to modify the region of 
video selected as the target of tfce ic caks dip* 1 operation. 

Figure 6 shows the result of typing a timecode into the text box and pressing the "seek* 
button 61. The video moves to the appropriate frame and the text is rendered in green to 
20 indicate a successful operation (i,e 4 the specified timecode could be found). 

Industrial Applicability 

i 

: As a simple example of the use of tie invention consider this scenario* An education 

25 professional is preparing mate ri als for a lecture they are about to give. A part of this is an 
electronic slide presentation with some of the slides containing embedded media objects. On 
running through the presentation they realise that it overruns the time allowance if all the 
. media clips wee played in their entirely. Using the system described here the media can be 
. quickly and efficiently trimmed to suit requirements, without the need for switching 

30 environments or applications. 
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Claims. 

What is claimed is: 

L A computer based system for manipulating digital media 3 the system adding the capability 
5 for media-manipulation to software media players, Si?ch. that this capability is intrinsic to 

the media player, and which comprises a sec of took that ensure that consistent 
behavioural, visnal and functional aspects are maintained between media player based 
applications, 

2. The system of claim 1 where media includes, but is not confined to, video a&d audio. 

10 3, The system of claims J & 2 where manipulation includes, but is not confined to, the 
operations of one or more ofi 

(a) Editing; trimming annotating; effects* transitions; appearance; presentation; 

4- The system of claims 1-3 comprising one ot mote of; 

{a) A software component that implements a cache fbt portions of the media file in the 
15 memory of the client machine, 

(b) A software component that implements a process equivalent to a state machine, 
whose cransidons guide a user through a sequence of interactions with a graphical user 
interface (GUT), 

(c) A software graphics component of the GUT, that implements visual feedback to the 
20 user of the current state. 

(d) A software graphics component of the GUI that implements a visual metaphor that 
provides the user -with an intuitive understanding of the operation of the interface. 

(e) A software graphics tenderer component that allows combination and/or overlay of 
graphical data for the GUI with pixels chat are decoded from the video part of the media 

25 file and rendered Into the video area on the screen. 

(£) A software component that implements an export of the pwicessed media to 
memory. 
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& A software component that implements me ability to read a description file® and 
construct playback in accordance with set instructions, or write such tottucdons from a 
current playback, 

<h) A software component of me GUI that allow* labels or triggers of various types to 
S be added to significant parts of me media file In order to identify item as such. 

5 The system of ckims 1 - 4 where labels Include, bat ate not conned to, edit in and out- 
point,, s hot boundaries, Group-Of-Picture (GOP) boundaries, closed-capdon text and 
rimecodc 

6, The system of claims 1 - 5 where the triggers include, but are no« confined to, initiate 
10 pop-ups, hold frames for a given duration, loop and messaging. 

7. The system of claims 1 - 5 ad d itionally comprising: 

A software decoder component that maps metadata contained in the media Sic to 
labels, where the meta-data includes, but is not confined to, shot boundaries, Group-Of- 
Pictutc (GOP) boundaries, closed-caption and dmecode. 
15 8, The system of claims 1 - 7 additionally comprising: 

A software agent component that maps aspects of the interactive behaviour of the user 
tow configuration information that may modify aspects of me behaviour of the GUI. 

9. The system of claims 1-8 addirionaDy comprising: 

20 A media file that may optionally be selected and played by the user, which provides 

instruction in the use of the GUL 

10. The system of claims 1 - 9 where the media player runs on a PC 

U. The system of claims 1 - 9 where me media player tuns on a set-top box. 
25 12. The system o£ claims 1 - 9 where me media player runs o a a Personal Video Recorder. 

13, The system of claims 1 - 9 where the media player runs on a mobile Personal Digital 
Assistant 

14. The system of claims 1 - 9 whetc the media player runs on a mobile phone. 



... . -m • ^^j-r^j iu;aitooai t +w+ F.SIB''^* 

11 

15. The Bystexa of claims 1-14 where die visual appearance of the GUI Is sensitive to the 
context in "which the user of d*e system is woririJOg such that the visual impact of the GUI is 
minimised when not needed. 

16. The system of claims 1-15 where the availability of die set of took Is sensitive- to the 
context in which the user of the system is working* 

17. The system of claims 1-15 whcie tfce context in which the user of the system id 
worfatig is determined by reference to the position of the saceen cursor with respect to the 
position of the video window. 



^w-cfcjckj j. r r kuti: ukj_13±n LUND UN +44 -5072038643 TO:016338i4444 P.019'0g4 



10 



n 



12 



File Viuw Play Favorites Go. Upgrade HeEp 




4- 4, r 



a'cb;^|bMysic.'.Sl Med, 



Hufdei: 



03:19/06:15 zHl 



Figure 1 



MOTTO EE AiwIEfc'nSD 




ji* rushes_ntsc_z,nsm - vvfndGfup 



Rle View Play • Favorites! c&p. upgrade 'Help • ' i 




... ,. , 



<0 



| Show: 


; Clip: 






; m\ Author: 






Copyright: 










03:02/06:15 'Mi 




Figure 2 





3 



i File View Play' Favorites Go 1 Upgrade. Hefa 



rushes_ntsc_z.n$nra - Windbf 



4- ■+ 




Radio A Musi.^ Q Mbtfe Guide I 



mm ' 



IP 



:00:00: 



n "h'l m *: »' m | b- I, ' :;. ^i^J^ 



Clip: 

£g Author: 
Copyright: 

Playinq 



03:02/06:15 =!>*= 



Figure 3 



NOT TO BE AMSNPED 



rushes ntsc z.nsm - Wimito; 



File View I Play- Favorites : Go Upgrade.;; Help 



Radio AMusKQ Media Guide- 




Playing 



04:33/06:15 



Figure 4 



52 



51 



Ha, rushes_ntsc_£ <nsm VJipd^^ 



Fife View- • Play ' Favorites go upgrade Help 



.i > 
• i it 




- 1. 1 -I %• 



* ,ir 1 

I 1 1 



41 r 



■Hi 4 — U 



Show: | 


Clip: 




iSj Author: 




Copyright: 




! Playing 


04:33^06:15 



53 



Figure 5 




TO: 01633814444 



P. 024^024 



File View ' Play, Favorites ' .Gjp Upgrade ,He|p; 1 ' ' ! 




fST ii ■ [ w 'tit \ m I ' '/ <$ j^j^j 




Clip; 
Author: 
Copyright: 



Flaying 



05:03/06:15 



Figure 6 



61 



MOT TO BE ATTENDED 



PCT Application 

PCT/GB2004/000305 




This Page is Inserted by IFW Indexing and Scanning 
Operations and is not part of the Official Record 

BEST AVAILABLE IMAGES 

Defective images within this document are accurate representations of the original 
documents submitted by the applicant. • 

Defects in the images include but are not limited to the items checked: 

□ BLACK BORDERS 

□ IMAGE CUT OFF AT TOP, BOTTOM OR SIDES 

□ FADED TEXT OR DRAWING 

□ BLURRED OR ILLEGIBLE TEXT OR DRAWING 

□ SKEWED/SLANTED IMAGES 

□ COLOR OR BLACK AND WHITE PHOTOGRAPHS 

□ GRAY SCALE DOCUMENTS 

□ LINES OR MARKS ON ORIGINAL DOCUMENT 

□ REFERENCE(S) OR EXHIBIT(S) SUBMITTED ARE POOR QUALITY 

□ OTHER: . 

IMAGES ARE BEST AVAILABLE COPY. 
As rescanning these documents will not correct the image 
problems checked, please do not report these problems to 
the IFW Image Problem Mailbox. 



